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DESCRIPTORS- ♦READABILITY i ♦CLOZE PROCEDURE! 



SYNTAX! 6RAMMAR! 



READABILITY RESEARCHERS HAVE MADE ADVANCES IN THE PAST 
FEW YEARS! INCREASING THE ACCURACY OF READABILITY FORMULAS BY 
AS MUCH AS 75 PERCENT. THIS PROGRESS WAS POSSIBLE LARGELY 
BECAUSE RESEARCHERS IN SEVERAL DISCIPLINES DEVELOPED RESEARCH 
TOOLS WHICH AIDED IN THE STUDY OF READABILITY. PSYCHOLOGISTS 
DEVELOPED THE CLOZE PROCEDURE INTO AN ACCURATE AND RELIABLE 
METHOD OF MEASURING LANGUAGE DIFFICULTY. LINGUISTS DEVELOPED 
DESCRIPTIONS OF VARIOUS FEATURES OF LANGUAGE! AND THESE 
DESCRIPTIVE DEVICES WERE ADDED INTO NEW TECHNIQUES FOR 
MEASURING THE FEATURES OF LANGUAGE THAT INFLUENCE ITS 



COMPREHENSION DIFFICULTY. FINALLY! ADVANCES IN THE 
UNDERSTANDING OF THE MATHEMATICS USED BY READABILITY 
RESEARCHERS LED TO IMPROVED DESIGNS FOR READABILITY FORMULAS. 
THE RESULTS OF THIS PROGRESS IS THAT! WITHIN A YEAR OF TWO! 
EDUCATORS WILL HAVE IMPROVED TOOLS FOR DETERMINING WHETHER 
INSTRUCTIONAL MATERIALS ARE SUITABLE FOR USE WITH THEIR 
STUDENTS. THIS PAPER WAS PRESENTED AT A MEETING COSPONSORED 
BY THE INTERNATIONAL READING ASSOCIATION AND THE AMERICAN 
EDUCATIONAL RESEARCH ASSOCIATION CSEATTLE! MAY 5! 1967). 
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Hew Data on Readability 
Cosponsored Meetings 

American Educational Research Association 
Promising Hex? Research Designs In the Field of Reading 
Friday, May 5, 11:00 a«m.-12:30 p.m. 

Reading and language specialists are avid students of readability research, 
for this research atten^ts to discover idiat makes language easy or difficult, to 
understand. Readability researchers study the correlations betx^een various 
features of language and the difficulty children have In understanding language. 
This provides the specialists with the Information they need to tailor Instruct- 
ional materials to fit the reading abilities of their students. It also pro- 
vides them with readability formulas by xdileh they can determine If commercially 
prepared materials are suitable for their students. Finally, by studying hoi7 
the many features of language Influence comprehension, readability research 
provides Insights Into the nature of the con^rehenslon process. Itself. 

The past few years have been rapid and someitfaat startling developments In 
readability research. For exan^le, the readability formulas available only three 
years ago could, at best, predict only 25 to 50 per cent of the variation we 
observe In the difficulties of Instructional materials. Today, we have not one, 
but two proto-type formulas, which are able to predict 85 to 93 per cent of the 
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variation* This rapresents a vary high laval of praclslon and an Inqsrovanant 
of from 35 to 75 par cant ovar tha valldltlas of oldar raadablllty formulas* 

Tha purposa of this papar Is to descrlba some of the results of this research 
and the efforts currently being made to forge our newly gained knowledge Into 
practical educational tools* 

Among the most Important events leading up to the present developments 
was the publication of two books summarizing the readability research done up 
to that time* One was by Chall (1958) and the other by Klare (1963). From 
these books It became clear that the future readability research had to con- 
centrate on three problems* First, a more reliable method had to be developed 
for measuring the difficulty children have In understanding materials* Sacond, 
researchers had to learn to measure and describe the linguistic features of 
' , ®sterlals that arc really In^ortant in affecting comprehension* Third, 
Investigators had to analyze their data In far more detail than they had up to 
that time* What follows Is an account of what resulted when efforts were made 
to attack each of these problems* 

Measurement of Coiq>rehenslou Difficulty 
Problem; Until recently. Investigators used multiple choice tests to 
determine the comprehen slon difficulties of materials* They made a test over each 
passage they were studying, tested the students after they had read each passage, 
and then found the mean percentage of questions answered correctly* The test 
means represented the difficulties of the passages* This method presented two 
problems* First, because the test was Itself a reading task, the Investigator 
was never quite certain whether he was measuring the difficulty of the passage 
or Just the difficulty of the test questions* Second, these tests could tell 



him nothing about how difficult each word, phraae, or sentence in the passage 
was* 

Construction of Cloze Tests : Shortly before Chall and Rlare published 

their books, Taylor (1955) reported his first work Td.th the close procedure. 

The cloze readability procedure can be used to make tests from any verbal 
instructional material. To do so, the Investigator selects the passage he 
wishes to study, deletes every fifth word, and replaces the deleted viords 
with underlined blank spaces of a standard length. The test Is given to 
children idio have not previously read the passage, and they are Instructed to 
write in each blank the word they think was deleted. Their responses are scored 
correct xdien they exactly match the words deleted, except that mlss-spelllngs 
are disregarded. 

Advantages ; Cloze readability procedure does not confuse the measurement 
of passage difficulty by Injecting an extraneous reading task Into the process# 

It also has the added advantage that Investigators could measure the difficulty 
of every \^rd, phrase, or sentence In a passage. 

Research * The cloze readability procedure Immediately drew the attention 
of readability researchers who set about studying cloze tests to see if they 
w er e valid and reliable measures of the comprehension difficulties of passages. 
This research has become far too extensive to review here. Bormuth (1967) and 
Ratdcln (1964) have each published detailed analyses of this research. In general 
the research showed that cloze readability tests are highly valid and highly 
reliable measures of the comprehension abilities of students and of the com- 
prehension difficulties of materials. 
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Description and Measurement of Language 

Early researchers felt a need to make their formulas so simple they could 
be used even by clerks having little technical knowledge of language* For 
example, to determine the complexity of a word, the clerk either counted its 
syllables or looked it up to see if it was on a list of words thought to be easy* 

To determine tlie grammatical complexity of a sentence, the clerk had only to 

/ 

count the number of words, and sometimes prepositions, in the sentence* l>Aiile 
it was, at that time, important for formulas to be sinqple, the old formulas 
vastly over-simplify the rich array of language features that influence its 
comprehension difficulty* The over- simplification also contributed to the 
fact that the old formulas were inaccurate* 

Vocabulary Complexity 

f 

Present investigators are probing more deeply into the question of what 
makes a word difficult to understand* It is not enough to say that the words 
on some list have been shovm to be easier to understand, for this leaves us 
still asking which of a i^ord's many meanings did children understand and why 
those words are easier for students* Nor is it practical to test all words 
directly on children^ especially Tdien we consider that most words have several 
meanings* What follows is a discussion of some of the features currently being 
investigated* 

Word Length : Children have always thought of long words as hard and short 

words as easy, and researchers have recently rediscovered this fact and begun 
investigating word length as a variable* Coleman <1961) found that a word's 
difficulty has a correlation of -*90 with both the nuad>er of letters and the 
nuod>er of syllables in the words* Bormuth (1966) found correlations of -*76 
and -*68, respectively, for the same measures* 
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Mtorphologlcal Complexity ; A word la often a complex structure which 
may be analyzable into a stem and a series of Inflectional, derivational, and 
lexical affixes* It seems that this is an important source of difficulty in 
understanding words* Coleman (1961) found that word difficulty had a correla- 
tion of -*88 with the number of affixes and stems into which a word could be 
analysed and a correlation of the same size with the number of inflectional 
morphemes* 

Abstractness ; Although there are almost as many meanings’ of the word 
abstractness as there are people who use it, nearly everyone agrees that, 
whatever it is, it has an influence on the difficulty of a word* Coleman (1966) 
devised a definition which permitted him to count reliably the nusd>er of nouns 
that referred to internal mental states and found that this number had a 
correlation of -*78 with passage difficulty* 

Frequency ; It has long been known that the frequency with which i. word 
is used has some influence on the difficulty people have in understanding it* 
But, frequency was thought to be a weak variable since Lorge (1949) hid found 
only a correlation of *51 between it and difficulty* Hore recently ^ Bormuth 
(1966) has shown that frequency and difficulty have a curvilinear relationship 
and that, when this fact is taken into account, they have a correlation of *66* 
Rlare (1967) has now taken a position that the frequency of a word may directly 
reflect most of the other characteristics of the word* 

Grammatical Complexity 

The degree of intricacy of the gramsatical relationships between the parts 
of a sentence has always been considered an important source of the difficulty 
in understanding the sentence* Until recently, the chief means of assessing 



er|c 



f 



page 6* 

grannatical con^lexlty consisted of counting the nusdier of words In sentences. 

Two najor objections can be raised to considering sentence length as the 
sole factor affecting gramnatlcal complexity. First, It forces us to accept 
the dubious proposition that all sentences containing the same number of words 
possess the same degree of complexity. Thus, we are asked to believe that the 
sentence The man saw the boy who fojSS^ penny which was lost has the same 
degree of con^lexlty as The penny \dilch the boy whom man saw found was losg. 
Second, the number of words In a sentence does not measure a natural unit of 
language. We cannot simply add or chop off a few words to make the sentences 

more or less complex. 

The grammatical complexity of a sentence actually results from the 
grammatical structure of the sentence. Consequently, modern researchers are 
Investigating measures of grammatical complexity based on the grammatical 
structures of sentences. This approach Is given firm support by the experiments 
performed by Martin (1966) and Johnson (1966a, 1966b) which demonstrate that 
people utilize the phrase structure of sentences as they process the sentences. 

Syntactic Depth : Yngve (1960) developed a measure of syntactic complexity 

which obtains the number of grammatical facts a reader must temporarily hold 
In his memory as he reads a sentence. Presumably, the more grammatical facts 
the reader must remember as he reads a sentence, the more likely he Is to forget 
one of those facts and the more likely he Is to fall to comprehend some aspect 
of the sentence. Bormuth (1964) and Martin (1966) have each shown that people's 
responses to sentences are closely related to the depth measures of the sentences. 
Bormuth (1966) found a correlation of -.55 between depth and passage difficulty. 
Further, he found (1963) that the effects produced by depth were Independent of 
those produced by sentence length. 
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Hodlfler Distance! A variation on th^ depth measure was developed by 
Bormuth (1967) tnd Is being Investigated by him and by Coleman and Aquino (1967). 
This variable measures the number of YM>rds occurring between a word or phrase 
It siodlfleB on the theory that the longer a grammatical fact Is held In memory, 
the more likely It Is that It will be forgotten. Preliminary results Indicate 
that tbere Is a correlation of -.80 to -.90 between this feature and passage 
difficulty. 

1 ransformatlonal Conm l^t^y; A sentence such as The little boy ran may 
be represented as resulting from a transformation which embedded the kernel 
sentence The boy was little Into the kernel sentence The boy ran. Chomsky (1965) 
has argued that to Interpret a sentence people must transform a sentence back 
Into Its kernel sentences. 

An Interesting aspect of the transformation analysis Is the fact that It 
can be used to measure what early researchers referred to subjectively as 
being the Idea density of materials. Coleman (1966) found that the number of 
nomlnallzed verb and nomlnallzed adjective transformations had correlations of 
-.76 and -.57, respectively, with passage difficulty. Many parts of speech 
represent transformations, also. Bormuth (1966) found that counts of the various 
parts of speech had correlations as high as .81 x^lth passage difficulty. His 
present studies are analyzing the effects associated with each of the trans- 
formations found In English. 

Contextual Variables 

Modern researchers are looking beyond the word and the sehtence to find 
the features of language that operate over longer segments of text to Influence 
comprehension. Rosenberg (1966) found Indications that passage^contalnlng words 
idilch people tend to associate i«lth each other are easier to recall. Coleman 
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and Aquino (1967) are finding that anaphoric analyus yield variables that 
predict passage difficulty* Anaphora arc repeated references to a concept 
In a passage* The use of anaphora Indicates the extent to vhlch a passage deals 
In depth with a single topic* Since the work In this area Is only beginning, 
it Is still too early to predict Its outcomes* But, It seems certain that 
gains In this area x^lll have great value In Increasing our ability to predict 
and control passage difficulty* 



Readability Formulas 

Early Investigators had to defer the Investigation of many Important 
problems until research In other disciplines had made tools available for 
studying those problems* As may be seen from the proceeding discussion, linguistic 
research provided readability researchers with new and powerful tools for analysing 
language* Similarly, research tools became available for studying the problems 
Involved In designing readability formulas* As a result, x^e have now learned 
enough to design much sounder readability formulas* 

Readability and Reading Ability 

A problem long plaguing researchers was the question of xdiether the features 
that Influenced readability for poor readers also Influenced the readability of 
materials for more able readers* If the same features of language Influence 
readability for both and by the same amount, then a single and fairly simple 
formula can be used to predict readability for all students, regardless of their 
level of accomplishment In reading* But if different features Influence difficulty 
for students of differing levels of reading achievement or If the same features 
Influence difficulty by different amounts, then X 7 c must develop more complex 
materially different kinds of formulas* Bormuth (1966) studied this problem 
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and found (*at, ragardleae of the person' a reeding ability, the same features 
of i«ne ii« e « that caused difficulty for hla caused the sasie amount of difficulty 

for others. 



I Shapes of the Relationships 

A second question was whether a given amount of Increase In a feature of 
; the language Increased difficulty regardless of boa much was already present. 
tot example, is the difference in difficulty between two and three syllable 
words as great as the difference in difficulty betoreen 7 and 8 syllable words? 

If not, the simple correlation techniques used by early researchers yield 

results. Bormuth (1966) found the differences were not always the 
same. Figuratively speaking, adding another syllable to a one syllable word 
increases its difficulty far more than adding another syllable to a seven syllable 
word. The same is true of many other features. Hence, future readability 

must Include appropriate transformations of measurements taken of these 

features* 



Form of the Formulas 

The traditional readability formulas are presented in the form of vdiat is 
called a multiple variable, linear equation. These equations have a characteris- 
tic that makes them unsuitable for use as readability prediction formulas. To 
use them, the researcher must assume that any correlation observed between tow 
variables, say sentence length and word length, must always exist. This simply 
is not true of the language features used in most formulas. The result is that 
the old formulas yield misleading results idienever the correlation is anything 
other than the correlation the formulas assume. Mbst future readability formulas 
will probably be designed to provide a profile of the level of difficulty 
represented by each of the language features in a passage, 
i 
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Summary 

Readability researchers have made rapid strides la the past few years» 
Increasing the accuracy of readability formulas by as much as 75 per cent. 

The reason lies largely In the fact that researchers In several disciplines 
have developed research tools which have aided greatly the study of reada- 
bility. Psychologists have developed the cloze procedure Into an accurate 
and reliable method of measuring language difficulty. Linguists have 
developed descriptions of various features of language and these descriptive 
devices have been further adapted Into powerful new techniques for measuring 
the features of language that Influence Its con^rehenslon difficulty. 



Finally, advances In our understanding of the mathematics used In our analyses 
have lead to improved designs for readability formulas. The result of these 
advances Is that, within a year or two, educators will have placed In t he ir 
hands powerful new tools for determining If Instructional materials are 
suitable for use xTlth their students. 
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